Skip to main content
Version: 1.0.0

populateVariantFromSku

Function Name: populateVariantFromSku

Author: Domenico Cerone Creation Date: 02/10/2025
Last Reviewer: Domenico Cerone

Trigger: HTTPS (onRequest)

Purpose: Complete workflow for automatic product data population from multiple catalog orders. Handles the entire process: order loading → product loop → validation → download → mapping → storage with public URLs. Processes all products from CatalogOrders and creates/updates both Products and Variants collections. Note: EAN/UPC codes are stored at variant level only, not at product level.

Detailed Functionality​

This Firebase Function performs a comprehensive 9-step workflow for processing catalog orders with dynamic batch processing optimized for volumes from 5 to 500+ SKUs.

1. CATALOG ORDER LOADING​

  • Receives catalogOrderId as input parameter
  • Loads document from 'CatalogOrders' collection
  • Extracts list_3d_assets_results with product array
  • Each element contains: skuCode, eanCode, upcCode, isSelected, result
  • Processes all products in list_3d_assets_results

2. DYNAMIC PARALLEL BATCH PROCESSING WITH MUTEX SYNCHRONIZATION​

Dynamic batch configuration based on SKU volume:

  • ≤50 SKUs: 10-element batches + 500ms pauses (fast mode)
  • 51-200 SKUs: 15-element batches + 1000ms pauses (balanced mode)

  • 200 SKUs: 20-element batches + 1500ms pauses (robust mode)

Processing Features:

  • Sequential batch processing to avoid Safilo API overload
  • Parallel processing within each batch for optimal performance
  • NEW: Mutex synchronization to prevent duplicate product creation → When 2+ variants of the same modelName are in the same batch → Exclusive lock system to serialize Products collection creation → Polling with backoff to wait for lock release (100ms → 2000ms max) → Automatic cleanup of expired locks to prevent memory leaks → Thread-safe tag creation using findOrCreateTagThreadSafe() from tagUtils.js → Prevents duplicate Tags documents in high-concurrency scenarios
  • Intelligent dynamic pauses between batches for system stability
  • Detailed progress logging with memory monitoring for orders >200 SKUs
  • Individual product and batch error handling without blocking others
  • Separate API call tracking for each product
  • Optimized for handling up to 500+ SKUs without timeout (60 minutes maximum)
  • Automatic modes: Standard, Optimized, Ultra-Robust

3. CHECKSKU API CALL WITH AUTOMATIC RETRY​

API Details:

For each product with robust retry system:

  • ALWAYS calls checkSKU API to get complete data (SKU, EAN, UPC)
  • Automatic retry system: 3 attempts per parameter (SKU, EAN, UPC)
  • Fallback logic: first all parameters, then one at a time
  • Progressive delay between retries: 50ms → 200ms → 400ms to avoid overload
  • Required to have UPC in mapped JSON
  • Robust product validation in Safilo system
  • Updates allProductData with all codes for current product
  • API call tracking in "apiCalls" object per product with retry status

4. SKU FOLDER CHECK FOR PRODUCT​

  • Checks existence of folder ARS_Library_Product_Data/{skuCode}/
  • If NOT EXISTS → proceeds with API download (STEP 6)
  • If EXISTS → proceeds with existing JSON check (STEP 5)
  • (SKU validation already completed in STEP 3)

5. ORIGINAL JSON CHECK FOR PRODUCT​

  • Verifies existence of file: ARS_Library_Product_Data/{skuCode}/json/{skuCode}.json
  • If EXISTS: → Retrieves metadata and token for public URL → Reads JSON content from Firebase Storage → Proceeds to mapping and document creation
  • If NOT EXISTS → proceeds to STEP 6 for API download

6. API GETMODELS DOWNLOAD WITH AUTOMATIC RETRY (if JSON doesn't exist)​

API Details:

Processing Features:

  • API call: spaarkly.getModels?sizeCode={skuCode}
  • Automatic retry system: 3 attempts with progressive delay (100ms → 300ms → 600ms)
  • Robust HTTP error handling (404, 500, timeout, network errors)
  • JSON response format validation with data verification
  • Temporary filesystem save with automatic cleanup
  • Firebase Storage upload with UUID token for public URL
  • Guaranteed temporary file cleanup even on errors
  • Detailed retry and success logging for debugging
  • Proceeds to mapping and document creation

7. DOCUMENT CREATION AND MAPPING​

Dynamic mapping system:

  • Dynamic parsing of mapping_template.xlsx from Firebase Storage
  • SAP metadata loading from Firebase Storage (metadata_files/)
  • Product image download (8 images: detail 00-07) from Safilo API
  • Data extraction from original JSON based on template rules
  • Automatic SAP codes translation
  • Special field handling: → SPECIAL_PRODUCT_DESCRIPTIONS: Creates separate language fields (description_en, description_it, etc.) → SPECIAL_IMAGE_NAMES: Extracts public URLs with tokens → specific posters (00,07,02,01) + list_images array
  • Mapped JSON save on Storage with public URL
  • Create/update document in Products collection (without catalogOrdersRef, no EAN/UPC codes stored here)
  • Create/update document in Variants collection (batch, catalogOrdersRef, and prioritiesModeling extracted from CatalogOrder, EAN/UPC codes stored here)

Related Components:

  • createProductDocument - Handles Products collection management with thread-safe tag creation
  • createVariantDocument - Handles Variants collection management with brand normalization
  • tagUtils - Provides thread-safe tag creation utilities with mutex synchronization to prevent duplicates

Brand Normalization System​

The function includes an intelligent brand normalization system that applies business rules to standardize brand names from SAP data to consistent marketing brand names.

Brand Mapping Rules:

  • Hugo brands: "HUGO" → "HUGO EYEWEAR"
  • Boss brands: "HUGO BOSS", "BOSS ORANGE" → "BOSS EYEWEAR"
  • Carrera brands: "CARRERA BY JIMMYCHOO", "CARRERA BIKE", "CARRERA SNOW", "CARRERA DUCATI" → "CARRERA"
  • Polaroid brands: "POLAROID ANCILLARIES", "POLAROID KIDS", "POLAROID STAYSAFE" → "POLAROID"
  • Smith brands: "SMITH FASHION & ACC.", "SMITH BIKE HELMETS", "SMITH SNOW", "SMITH BIKE GOGGLES", "PRIVATE LABEL SMITH", "SUNCLOUD" → "SMITH OPTICS"
  • Other brands: Remain unchanged

Implementation: The mapToNormalizedBrand() function is called automatically during data mapping to ensure consistent brand naming across both Products and Variants collections.

Benefits:

  • Consistent brand representation across all collections
  • Simplified brand management and filtering
  • Alignment with marketing brand guidelines
  • Automatic application without manual intervention

Mutex Synchronization System​

Problem Solved: When multiple variants of the same product (same modelName) are processed in parallel within the same batch, they could attempt to create the same Products document simultaneously, leading to race conditions and potential duplicates.

Solution: Exclusive lock system using Map-based mutex:

  • Lock Acquisition: Before creating/updating a Products document, the function acquires an exclusive lock using the modelName as key
  • Lock Key Format: PRODUCT_CREATE_\{modelName\}
  • Polling Mechanism: If a lock is already held, the function waits with exponential backoff (100ms → 2000ms max)
  • Automatic Cleanup: Expired locks (older than 10 minutes) are automatically cleaned up to prevent memory leaks
  • Granular Locking: Each modelName has its own lock, so different products can be processed simultaneously
  • Error Safety: Locks are released in finally blocks to ensure cleanup even on errors

Benefits:

  • Prevents duplicate Products documents
  • Maintains data consistency in high-concurrency scenarios
  • Optimizes performance by allowing parallel processing of different products
  • Provides automatic recovery from stuck locks

Variants Collection Fields Population​

The function populates the following fields in the Variants collection with data extracted from the CatalogOrder and product processing:

Base Fields:

  • skuModel ← skuModel (primary identifier)
  • eanCode ← eanCode
  • frameColor ← frameColor
  • lensesColor ← lensesColor
  • glassesName ← glassesName
  • mainBrandRef ← ID brand (searches/creates in MainBrands collection)
  • upcCode ← upcCode
  • batch ← batch (extracted from CatalogOrder)
  • productRef ← ID of parent Products document
  • catalogOrdersRef ← ID of CatalogOrder that generated the variant
  • prioritiesModeling ← priority (extracted from product in list_3d_assets_results)

Image Fields:

  • poster ← poster (view 00 - main frontal)
  • poster2 ← poster2 (view 07 - side angle)
  • poster3 ← poster3 (view 02 - frontal detail)
  • poster4 ← poster4 (view 01 - alternative angle)

Size Object (all fields as strings):

  • size.bridge ← bridge
  • size.frameWidth ← frameWidth
  • size.lensHeight ← lensHeight
  • size.lensWidth ← lensWidth
  • size.size ← size
  • size.templeLength ← templeLength
  • size.frontHeight ← frontHeight
  • size.hingeToHinge ← hingeToHinge
  • size.lensCircumference ← lensCircumference
  • size.lensAngle ← lensAngle
  • size.phantoscopicAngle ← phantoscopicAngle

Timestamp Fields (as JavaScript Date objects):

  • lastUpdate ← current timestamp (always updated, Date object)
  • skuReleaseDate ← skuReleaseDate (if present in mapped JSON, Date object)
  • skuEndDate ← skuEndDate (if present in mapped JSON, Date object)
  • advStartDate ← advStartDate (if present in mapped JSON, Date object)
  • advEndDate ← advEndDate (if present in mapped JSON, Date object)

Tags Arrays:

  • list_size_color_tags ← [frameColor tag ID, size tag ID] (searches/creates in Tags collection)
  • list_images ← list_images (array of objects with view and url of images)

Brand Normalization: The mainBrandRef field is automatically normalized according to business rules:

  • Hugo brands: "HUGO" → "HUGO EYEWEAR"
  • Boss brands: "HUGO BOSS", "BOSS ORANGE" → "BOSS EYEWEAR"
  • Carrera brands: "CARRERA BY JIMMYCHOO", "CARRERA BIKE", "CARRERA SNOW", "CARRERA DUCATI" → "CARRERA"
  • Polaroid brands: "POLAROID ANCILLARIES", "POLAROID KIDS", "POLAROID STAYSAFE" → "POLAROID"
  • Smith brands: "SMITH FASHION & ACC.", "SMITH BIKE HELMETS", "SMITH SNOW", "SMITH BIKE GOGGLES", "PRIVATE LABEL SMITH", "SUNCLOUD" → "SMITH OPTICS"
  • Other brands remain unchanged

Products Collection Fields Population​

The function populates the following fields in the Products collection with data extracted from the product processing:

Base Fields:

  • modelName ← glassesName (primary identifier)
  • age ← catAgeForGender
  • eanCode ← eanCode
  • manufacturedProductStatus ← manufacturedProductStatus
  • rxAble ← rxAble
  • mainBrandRef ← ID brand (searches/creates in MainBrands collection)
  • upcCode ← upcCode

Image Fields (from current variant):

  • imgUrl ← poster
  • thumbnail ← poster
  • urlImage ← poster

Timestamp Fields:

  • lastUpdate ← current timestamp (always updated, Date object)
  • productReleaseDate ← productReleaseDate (if present in mapped JSON, Date object)
  • productEndDate ← productEndDate (if present in mapped JSON, Date object)

Multilingual Descriptions:

  • descriptionEn ← description_en
  • descriptionIt ← description_it
  • descriptionEs ← description_es
  • descriptionFr ← description_fr
  • descriptionDe ← description_de

Tags Arrays (aggregated from all variants):

  • list_frame_color_tags ← [frameColor from all variants of the product]
  • list_size_tags ← [size from all variants of the product]
  • list_line_tags ← [nomeLinea]
  • list_tags ← [catType, catAgeForGender, catGender, catForma, catMaterialOne, polarised, catLensesTreat, hingeType]

Variants Management:

  • variants_map ← array with entry for each product variant:
    • eanCode, id, publishedInARSLibrary (""), skuCode, upcCode, variantName (glassesName)
  • list_variants ← [ID of all product variants]
  • availableVariants ← always 0 (logic managed externally)

Input Data Structure​

The function expects the CatalogOrder document to contain the following structure:

Required Fields:

  • clientId: ID of the client
  • list_3d_assets_results: array of products with skuCode, eanCode, upcCode, isSelected, result, priority
  • orderName: name of the order
  • profile: client email
  • quoteResponse: response status (e.g., "accepted")
  • batch: batch identifier for variants

Product Structure in list_3d_assets_results:

{
  "skuCode": "20637008650QT",
  "eanCode": "0762753916013",
  "upcCode": "762753916013",
  "isSelected": true,
  "result": "Available",
  "priority": "High"
}

Priority Values: The priority field from the product is used to populate the prioritiesModeling field in the Variants collection. Common values include:

  • "High" - High priority for modeling
  • "Medium" - Medium priority for modeling
  • "Low" - Low priority for modeling
  • "" (empty) - No specific priority

8. CATALOG ORDER UPDATE​

  • Updates list_3d_assets_results with productRef and variantRef for each processed product
  • Maintains all other catalog order fields unchanged
  • Sets status = "working_3d_assets" at operation end

9. PROFILES UPDATE​

  • Adds catalogOrderId to list_3d_catalogOrders arrays of appropriate profiles
  • Requester profile: extracts 'profile' field from CatalogOrder
  • Admin profiles: searches all profiles with role = 'Admin'
  • ModellerSupervisor profiles: searches all profiles with role = 'ModellerSupervisor'
  • Duplicate verification: doesn't add if catalogOrderId already present
  • Robust error handling: continues even if some profiles fail
  • Final response with all processed product results and profile updates

Input (Payload)​

Method: POST/GET

Headers:

  • Content-Type: application/json (for POST requests)

Body (POST) or Query Parameters (GET):

{
  "catalogOrderId": "VHNGMFgVezqLDrXBv4iV"
}

Parameters:

  • catalogOrderId (string, required): Unique identifier of the catalog order to process

Output (Success)​

{
  "success": true,
  "catalogOrderId": "VHNGMFgVezqLDrXBv4iV",
  "totalProducts": 3,
  "successCount": 2,
  "errorCount": 1,
  "processedProducts": [
    {
      "success": true,
      "skuCode": "20637008650QT",
      "eanCode": "0762753916013",
      "upcCode": "762753916013",
      "productRef": "auto_generated_product_id",
      "variantRef": "auto_generated_variant_id",
      "productOperation": "create",
      "variantOperation": "create",
      "message": "Product 20637008650QT processed successfully",
      "apiCalls": {
        "checkSKU (EAN, UPC, SKU)": "https://...spaarkly.checkSKU?sku=...",
        "getModels (file .json)": "https://...spaarkly.getModels?sizeCode=...",
        "getImages (thumbnail images)": "https://safilo-pd-cdn002.safilo.com/... (or '(cached)' if existing)"
      }
    }
  ],
  "profileUpdateResult": {
    "success": true,
    "totalUpdated": 5,
    "totalErrors": 0,
    "details": {
      "requesterProfile": \{ "updated": true, "error": null \},
      "adminProfiles": \{ "count": 3, "updated": 3, "errors": [] \},
      "supervisorProfiles": \{ "count": 2, "updated": 2, "errors": [] \}
    }
  },
  "message": "Processing completed for CatalogOrder VHNGMFgVezqLDrXBv4iV: 2/3 products processed successfully",
  "recap": "CatalogOrder VHNGMFgVezqLDrXBv4iV loading → 3 selected products → 2 products processed successfully → 1 products with errors → CatalogOrder update completed → status set to working_3d_assets → profile updates: 5 profiles updated",
  "allApiCalls": {
    "Product_1_20637008650QT": \{ ... \},
    "Product_2_1095638075515": \{ ... \},
    "Product_3_INVALID_SKU": \{ ... \}
  }
}

Technical Specifications​

Firebase Function Configuration:

  • Timeout: 3600 seconds (60 minutes) - maximum for HTTP Functions v2
  • Memory: 2GiB - optimized for extreme volumes (300-500 SKUs)
  • Region: europe-central2 - optimal performance for Europe
  • CORS: Enabled for frontend calls

Performance Guarantees by Volume:

  • 5-50 SKUs: 15 seconds - 2 minutes (standard mode)
  • 51-200 SKUs: 3-12 minutes (optimized mode)
  • 201-500 SKUs: 15-35 minutes (ultra-robust mode)
  • No timeout guaranteed up to 500 SKUs

Testing​

URL (if HTTPS): http://127.0.0.1:5001/arshadesstaging/europe-central2/populateVariantFromSku

Test with Emulator:

  1. Start Firebase emulator: firebase emulators:start --only functions
  2. Ensure you're using Node.js version 20: nvm use 20
  3. Test with curl:
curl -X POST "http://127.0.0.1:5001/arshadesstaging/europe-central2/populateVariantFromSku" \
-H "Content-Type: application/json" \
-d '{
  "catalogOrderId": "VHNGMFgVezqLDrXBv4iV"
}'

Postman Testing:

  • Method: POST
  • URL: http://127.0.0.1:5001/arshadesstaging/europe-central2/populateVariantFromSku
  • Headers:
    • Key: Content-Type
    • Value: application/json
  • Body: raw JSON (see examples above)

Deploy Command​

firebase deploy --only functions:populateVariantFromSku

Production URL​

Live Function: https://europe-central2-arshades-7e18a.cloudfunctions.net/populateVariantFromSku